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OBJECTIVES 

The process of evaluation was condemned by Cuba 
(1973) as a failure. Several symptoms were described by 
which this failure was recognized, including avoidance, 
anxiety, immobilization, misadvide, and no significant 
differences.- But perhaps the most important indication 
of failur'" was vague guidelines or no guidelines at all 
for educational evaluation/ 

Fortunately, this situation has improved somewhat 
over the past few years as a result of efforts by Steele 
(1973), Stuff lebeam (19-73), Grobman (1970), Roberson 
(1971), Klein, et.al. (1971), and Stake (1975), among 
others. Evaluation is now generally considered to include 
the processes of gathering and. analyzing information for 
the purpose of making decisions about programs and 
procedures These processes are organized according to a 
paradigm which usually includes several components, such 
as needs assessment, program planning, evaluation design, 
program monitoring, progress evaluation r and outcome 
evaluation . 

?V review of ESEA Title Ill-supported arts in 
education projects in Pennsylvania over the past five 
years , however , provides convincing evidence that the 
effectiveness of each has been lessened by the application 
of diverse and often irrelevant criteria for evaluation. 



.,As public funds for education have become more limited, 
austere measures have been and are being implemented to 
account for and -to justify these expenditures. 
Accountability and justification are determining several 
of the decisions which must be made concerning educational 
programs and projects • Evaluation processes must be 
..-.ocuted that will allow these decisions to be made in 
the best interests of today's children. 

Several arts in education projects have been funded 
through ESEA Title IV over the past two years in the 
Cominonwealth of Pennsylvania which collectively have the 
potential for. significant impact upon Pennsylvania - 
education. Current evaluation practices recommend 
initiating a statewide effort to identify and to use 
appropriate methods for monitoring and assessing the 
effectiveness of individual programs as well as their . 
collective impact. The purpose of this paper is to 
describe a cooperative effort to analyze and to translate 
the program objectives for several arts in education 
projects in such a way as to provide a base for independent 
yet concurrent and interrelated evaluation activities. 

B. PERSPECTIVE 

Bond and Dykstra (1967) presented a report of the 
Coordinating Center of the Cooperative Research Program 
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in First Grade "Reading Instruction. Common experimental, 
guidelines were followed in 27 individual studies 
comprising the Cooperative Research Program. Throe basic 
questions were posed which the Cooperative Research/ 
Program hoped to answer: 1) What characteristics were ' 
related to first grade reading and spelling achievement? 
2) Which program produced superior reading and spelling 
achievement at the end of the first grade? 3) Was any 
program more or less effective for students at different 
levels of reading readiness? Data were compiled from 
each of the individual studies relevant to these th::ee 
questions concerning community, school, and student: 
characteristics and achievement, teaching strategies and 
achievement, and program effectiveness and reading 
readiness. 

Each of the 27 studies was a complete study in 
itself. The program's unique characteristic was that 
each project director, in addition to carrying out his or 
her own analysis, made data available to a Coordinating 
Center, thus enabling analysis across individual projects. 
Therefore, common procedures for data collection and 
analysis as well as common experimental procedures were 
established during initial program planning. The 
Coordinating Center was charged to maintain communications 
and to collect, organize, analyze, and interpret the data 
common to each child in all 27 individual projects. 



In analyzing the Cooperative Research Program^ 
Grobman (1970) noted that it did not work perfectly. There 
were differences in procedures, materials, treatments, 
samples, and rigorousness of implementation between the 27 
projects than what had -been originally planned. However , 
the comprehensiveness of the data provided and the 
appropriate^ness of comparison groups throughout the 
Cooperative Research Program were indeed impressive. - 

This initial landmark study dest^ribed a relatively 
new type of research effort which Grobman suggested would 
be suitable for science education research. While 
recognizing the limitations of fewer, centralized, large-- 
scale research projects as well as those of many smaller 
isolated projects, cooperative, parallel, multi- 
investigator approach could take advantage of those 
benefits possessed by both while minimizing their 
limitations . 

This support for a cooperative parallel research 
effort applies equally to a cooperative parallel 
evaluation effort^ and could be considex^ed even more 
critical, for the results of evaluation ean determine 
future allocations for research and relaited activities. 
Although similarities exist between rese^arch and 
evaluation, their processes are not conterminous. 
However, the needs for cooperative paraULel evaluation 



activities could be considered as having the same bounds 
as those identified 2or cooperative parallel research. 

Just what is meant by the concept of cooperative 
parallel evaluation? Sharing thoughts and techniques for 
the common benefit of all is the essence of cooperations- 
parallelism refers to similar activities being conducted 
at the same time at two or more project sites; evaluation 
is a systematic procedure for collecting and analyzing 
information for the purpose of decision-making. 
Cooperative parallel evaluation, then, is a systematic 
procedure carried out at different project sites to 
gather information appropriate for answering questions 
and making decisions about areas of mutual concern. 

The potential for several benefits exists in a 
cooperative parallel evaluation project. One is the 
involvement of competent personnel from different regions 
and projects. Others include maintaining close contact 
with classrooms, teachers, and students across several 
school districts, bypassing the funding and administrative 
problems associated with single, large-scale projects, and 
standardizing evaluation efforts throug) planning and 
. using common purposes and common results to encourage 
comparability of findings from different projects. 

C. METHODS 

The convergent methods of the more generally accepted 
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prescriptive evaluation models/ when applied to the 
divergent processes of the arts, may generate an 
incongruous situation. Stake and Hoke (1976) recognized 
the need for different methods for evaluating the arts 
that wc ..xd be more compatible with divergent arte 
processes- They described responsive evaluation as having 
a qualification data base which depended upon their 
inability to tear down their own conclusions, rather .than 
depending solely upon a quantitative data base for 
determining program success or failure. 

Both qualitative and< quantitative data bases were 
recognized as components of an evaluation system for the 
initial organization of the cooperative parallel - 
evaluation of the arts in education projects. While some 
behaviors can be trans'l-ated into numerical data, there 
are others involving emotions, attitudes, and aesthetics 
that cannot be as easily transformed. This qualitative 
component of the evaluation project was assigned to each 
project director and became one of the responsibilities 
of the individual evaluation studies. It was not 
considered: further in the organization and implementation 
of the cooperative parallel evaluation. 

The basic model, employed for the cooperative parallel 
evaluation (CPE) was that developed at the UCLA Center for 
the Study of Evaluation (Klein, et.al, , 1971). This model 
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• consists of five phases — needs assessment, program 

planning, implementation evaluation, progress -evaluation, 
and outcome evaluation. Figure 1 illustrates this ^odel 
as adapted for the CPE project. Following the needs 
assessment phase, which was determined largely by a 
state department of education analysis of the proposed 
evaluation activities of several arts in education 
projects approved for funding through ESEA Title IV, 
Part C, further analyses of the projects by a planning 
group identified coiumon purposes; and objectives. The 
planning group included a state department specialist, 
a consultant from the Central Susquehanna Intermediate 
Unit, and one of the project directors. In addition, a 
planning conference for project directors was identified 
for the completion of the program planning, phase • 



IN'SERT FIGURE 1 ABOUT HERE 

The dominant component of the model synthesizes the 
conceptual schema of the cooperative and parallel 
evaluation activities. The common activities are 
coordinated in eight locations which' represent the eight 
separately funded arts in education projects.. This 
coordination is accomplished by a coordinating center 
which serves to catalyze the implementation of the CPE 
.evaluation activities. 
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' The' implementation and progress evalu"atio|i phases 
and the outcome evaluation phase correspond to the UCLA 
model- "The final corftponeht of the CPE model is the 
identification of recommendations and suggested solutions 
to commonly perceived problems in arts "in education 
activities. 

The Pennsylvania Title IV-C^project approval process' 
for FY 1976 required project starting dates to be 

scheduled during tlie late winter and early spring- of 1976 , 

\ 

well into the 1975-76 school year/ Such" a late start 
prevented meaningful implementation in the schools but 
did allow and encourage further planning . During this' 
time, the eight independently^ operating project directors 
voluntarily agreed to form an informal consortium ^for 
evaluation. The initial planning conference of project ^ 
directors was convened in late February, 1976, to share, 
plans and activities of the individual pro j'fects , to 
promote the need for cooperation and communication 
between projects, and to introduce tlie concept of 
cooperative parallel evaluation and/def ine the CPE model. 

A second meeting of directors was s-tcheduled for late 
Marchf 1976, to satisfy three purposes.- First, the 
concept of cooperative parallel evaluation was reviewed 
as an effort to organize a* sharing^ advising/ and assisting 

assessment project for the common benefit, of the eight 
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arts in education projects. Second, specific purposes 
and procedures for the CPE project were designed • Third; 
.a personal commitment from each project ^director was 
obtained to implement the CPE project. 

While the first meeting emphasized ' the affective 
dimension of group dynamics, conse/xsus generation, and 
cooperation as an attempt to build c/roup' morale , loyalty, 
and identity with the CPE "^pro ject , the second meeting 
concentrated on evaluation design and specification of 
independent and dependent variables for the overall 
project. For the remaining two months of the school 
year, the project directors were charged with revising 
the evaluation activities of their ■ individual projects 
to ensure systematic and valid assessments that would be 
compatible with the CPE model. 

At the State* Education Department • level , arrangements 
were made by the administrators of the Title IV-C program 
to coordinate the evaluation activities of independent 
state evaluators to allow them to meet and discuss the 
common successes, failures, problems, and concerns of the. 
^individual projects. Although hot part of the CPE project 
this activity was compatible; with the purposes of the- 
project; \ . ^ 

The Coordinating Center for the CPE project was 
established at the Central Susquehanna Intermediate Unit. 



and was charged to maintain communications and facilitate 
cooperation between the various projects and to collect 
and analyze common data collected from the individual 

/ . 

projects. It also served as the focus for evaluation j * 
consultation and CPE project design. During the summer 
months of 197''6, the Coordinating Center was rSs^^onsibLe 
for the iefinition of those independent and dependent 

0 

variables common to all projects and identified during 
the second meieting of project direqtors. It prepared 
operational def ihi tions of thesis variables and 
instrumentation to collect data from each proj^^ict as "* 
discussed at the planning meetings. 

A paoket of information including instructions ^ 
operational definitions, ev'aluation forms, and tests was 
sent to each project director in early fall at the 
beginning of the 1976-77 school year. Each project site 
was also visited by a member of the CPE planning group 
to. review the CPE project^- answer questions about the 
information packet, and seek clarification of the 
evaluation activities of each project. 

As much as possible, each project director was asked 
to. select a 'Comparison class (or classes:.:) or a comparison 
school building (or. buildings) . The stuidents and teachers , 
identified ^s a comparison group in the individual 
projects were matched by the project dinector as closely 

' - ' :' . • . • 




as possible tq^ students and teachers participating in the 
local project. ( 

Pretests were admini,' nd of October, 

19''6, and posttests were y.i. May, 1977. Each 

project director planned 125 instructional days between 
pre and posttesting for \. .e local project. 



\ 
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DATA SOURCE 



Three basic questions of concern to the CPE project 

were: 

1) To what extent are various student, teacher, 
Glass, school, and community characteristics 

- re^ajted to student progress in the arts? 

2) Which of the many approaches to arts-in-education 
produces more effective progress? 

3) Is any project uniquely effective or ineffective 
for students and/or teachers? 

Analysis of the eight Pennsylvania arts-in-education 
projects identified "teacher inservice" and "artists in 
classrooms" as the independent variables for this 
cooperative evaluation. Also identified have been 
"heightened self -awareness" and "alternative teaching 
"models". "as dependent variables-^ Further interest was 
sh6wn in a survey to suggest the' degree of relationship 
between arts-in-education activities and' achievement in 
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basic skills. 

Thus, the CPE project was designed to collect data 
from two dependent variables resulting f-om the 
manipulation of two separate indepencli m uiables. The 
independent variables were, defined as to-uxows: 

a) Teacher Inservice . — A planned series of sessions 
organized around various art areas and designed 
to improve the competencies of classroom teachers 

b) Artists in Classroonts , — A series of sessions 
planned by trained artists and prepared teachers 
to utilize arts processes to expand student 
learning and teacher instructional options. 

Operational definitions and descriptions of these 
variables as implemented in each local project were those 
prescribed by the original project proposals. 

Operational definitions for the two dependent 
variables were prepared with assistance from the 
Instructional Objectives Exchange (1972). These 
definitions follow : 

a) Self Concept. — 

1. When given the opportunity to display 

classwork, students will give evidence of 
positive self concepts by voluntarily 
posting their work. 
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2. Given a contrived situation i.n which the 
teacher describes a group of fictitiously 

-If 

esteemed students, class members will 
demonstrate positive self concepts by 
voluntr identifying themselves as 

stu ^ .jelong to this group. 

Instructional Strategy. — 

1. Teachers participating in an Arts-in-Education 
project (inservice and/or artists-in- 
classrooms) will perceive more positive ^. 
teacher-student interaction and growth in 
teacher-student relationships through 
professional^ analysis and self-evaluation of 
their own teaching behaviors. This analysis 
and evaluation will describe a significant 
increase in both the quantity and quality of 
interactions and relationships over those 
described by teachers not participating. 

2. Teachers participating in an Arts-in-Education 
project (inservice and/or artists-in- 
classrooms) will perceive an increase in the 
use, of a greater .variety of instructional 
strategies and a decrease in the number of 
student instructioi2Al problems and difficulties 
resulting from individualization of 
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instructional activities. These perceptions 
will be of a greater magnitude than those of 
non-participating teachers . 

■3. Teachers participating in an Arts-in-Education 
project ( inservice and/or artists--in- 

will demonstrate he invcj^vement 
• of other adults, outings into the comiTiunity, 
and flexible scheduling of activities with 
greater frequency than those tescTJers not 
participating, • 

4. Teachers participating in an Arts-in-Education 
project (inservice and/or arti? - -in-- 
classrooms) vill provide and ir^ itain a more 
attractive anc timulating cla. 
environment th n non-participat teachers. 
This will be determined by the .jer and 
variety of visual and manipulative materials 
present, the physical » arraiagement of 
classroom furnishings, and the frequency of 
changes in materials and fiarnishings . 

The c. n source for ea:a of these "Variables 
consis-ed 'j) i battery of definition f oinms , tests-, and 
eyaluatior . rms which were designed to (obtain information 
from local project staff,, teachers, and ;students. The 
total Fample for the CPE project, included all teachers 
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and students participating in each of the eight local 
projects. This sample included participants from rural, 
suburban, and urban areas, large and small school 
districts, and all grade levels. Data collection was 
pla^nned according to a four-dimensional matrix for 
analysis by project, setting, size of school, and grade 
level. 

RESULTS AND CONCLUSIONS 

Th::^ tir;s table for the CPE project was such that the 
first year v pl^ nned to r^ovelop and debug the CPE 
system." Ir r tion ant progress evaluation were 

planned as r lasis of the second year, while outc-me 

evaluation ^ .ned for the third. 

Now at t i.e . inning of the second year, of the 
project, M--'- .r.crete data has yet been collected from 
students a: ■ lers. This data is now being collected 

in the sepa. local sites. Observations, however, have - 

been obtaii ncerning the planninc design, and initial 

Ir^plementat the, cooperative parallel evaluatipn 

model, and vj-.rrl tentative conclusion : can be made. 

1^ All eighi Titla. IV-C arts in education projects 
in tn< ate parti ^ipated in the initial phases 
of /reject. The procedures for the CPE 

pio;c /ere accepted by all participants as 
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real, ongoing,^ and sensible. 

The CPE model was implemented in a cost efficient 
mvmner. Reallocation of evaluation monies at the 
local level were made in response to requests for 
modification of local activities to make the 
separate evaluation plans more compatible with 
the CPE project. A modest subr^istence grant of 
$1,250.00 from one of the projects was used to 
establish^ the Coordinating Center and form a 
loose confederation of arts in\ education projects 
across the state. State Department of Education 
personnel were able to more effectively manage 
their time since specialists coul^ monitor 
several projects in the same amount of time 
previously scheduled for one . 

Five project directors designed and. implemented 
revised local evaluation procedures as a result 
of competencies gained or activities suggested 
at the initial director planning meetings. 

The independent state evaluator^ were convened 
in a joint session to discuss their observations 
of the local projects and the CPE project. One 
of their recommendations was to continue the CPE 
project with greater suppor'^. irom the state's 
Title IV Office- 



is 



5. One of the pre -acts withdrew from the 

coif ederation as it re-emphasized a community 
education component and de-emphasized its arts 
in education component. This change was 
unrelated to the CPE project, however. n 

6- A teacher strike in one district and reluctr 

of staff to use some of the developed 5. truments 
in another delayed implementation of the CPE 
model by two local projects. All local projects 
have faced some interruptions since renewal 
procedures for local project grants had been 
changed somewhat forcing the project directors 
^ to spend £^ome extra time with the grants ; 
processes. . .. 

7. While implementation costs were modest, 

maintenance costs for the CPE project have thus, 
far exceeded original expectations . Maintenance^ 
costs include expenses required for regular 
communications between all projects, staff costs 
for the Coordinating Center, planning group, and 
^state monitoring, and workshops for project 
directors. ' . 



F. IMFTRTANCE 



Evaluation has generally not been well understood 
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by school people, Cuba (1973) noted that the lack of .an 
adequate definition , lack of theory, lack of criteria, 
and lack of trained personnel have plagued evaluation 
activities. One of the important benefits of this 
project has been the inservice traini^- pr- cal 
jc-.ool -zidople. Since evaluation is a requirement of each 
local project, the directors have been able to revise 
their iocnl evaluation activities to account for a common 
definition and model for evaluation. Suggested plans and 
procedures for systematic evaluation were provided by 
personnel trained in evaluation processes* 

There are several orher potential and actual 
advantages to cooperative parallel evaluation procedures. 
Evaluation efforts can be standardized across several 
projects, -chus providing a larger sample from which 
generalizations can be mad- . Competent personnel from 
different locations can be involved — the interchange of 
ideas by evaluation specialists working with local 
.projects can help strengthen the evaluation components 
of otaer local projects as well as the CPE project. 

In addition, direct comparisons can be made of 
different treatments through the CPE project that a . 
local project ::ould not accomplish for lack of 
participants End staff. Different methods of teacher 
inservice r for example, or Inservice for teachers in 



different academic areas could be investigated through a 
CPE project. A local project would probabl^' I: havr 
enough teachers for nrlpan ' ■ xq^.'^.e.s of bot i ' 
treatment groups, nor would it have t-e instruc-ionai 
resources to plan and present differer" inservice courses 
in different ways, whereas these coul"? be accomplished 
through a coordinated sharing of rescu^ces . through 
cooperative parallel evaluation. 

Another advantage is the maintenance of the integrity 
of local projects and encouragement of the completion of 
unique studies. While the CPE study has identified a few 
general concerns across all local pro j ects / each local 
project still remains an independent study- to implement 
the activities originally prescribed in its proposal to 
satisfy locally determined needs. The CPE project has 
not arrogated or altered any emotivities in local projects, 
with the exception of evaluation activities which, as 
previously mentioned, were revised as a result of the 
evaluation workshops for project directors. 

According to Grobman (1970), single;, large-scale 
projects prevent close contact between teachers and 
investigators because of an extended geographic area 
cover i by the study. The director of suet a project 

find it di..,-iicult to manage the details of 
implementation and would have considerate responsibility" 
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for administral In addition ant oik 

becomes more scarce, c la o: agencies ire seeking 
funding; large-scale projects are becoming more difficult 
to finance, especially in a single geographic area. The 
cooperative parallel study based upon a confederation of 
sTiialler independent projects avoids these problems. A 
local project director can maintain close contact with 
implementation and not be overburdened with administrative 
chores. Also, grant awards to several dif ferer regions 
is more politically feasible. 

LiiTitations of such an organization must be 
recognized, too. The cooperative parallel research 
project reported by Bond - and Dykstra (1967) was 
originally designed as such; the cooperative. parallel 
evaluation project reported herein .was not. The CPE 
project was planned and initiated after^the approval and 
negotiation phases/of Title IV-C funding had been 
completed.' None of the local project directors had an 
awareness of the CP^ project, much less included plans 
in their proposals for such activity di::ring local 
planning. Nevertheless, it is to the credit of the 
individual project directors that their flexibility and 
adaptability for revising their local projects permitted 
the CPE concept to progress. 

Other limitations of this stud:; are related to his 
lack of initial planning; to some degree these limitations 

22 
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include two-way coininunicatlons , financial commitment for 
maintenance of the model, and joint planning of the 
instrumentation to be used. 

The first phase of the project:, that of planning and 
design, has been completed. The advantages of the CPE 
project appear to overcome its limitations. It remains t 
be seen, however, whether the project will be successful 
in addressing the three questions listed in Section D. 
The data collection and analysis/iresulting from Phase II 
(Implementation and Progress Evaluation) will allow 
further judgement of the advantages and significance of 
cooperative parallel evaluation. i 
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Task Force Analysis 
of approved IV-C 
Arts in Education projects 



Figure 

COOPERATIVE PARALLEL EVALUATION 
PROJECT FOR 
mS IH EDUCATIOS 
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